Repository navigation
Ignore heading anchor ids in class/id rules - #19
Merged
Merged
Conversation
Sites build heading ids from the heading's text ("h-what-is-social-media",
"travel-related-coverage-2"), and docs generators put the same id on the
section the heading opens. The furniture, secondary and negative-weight
rules read words such as "social", "sponsor", "related" or "media" in those
ids and dropped real headings and sections. Such ids are now marked once in
the preprocessor and left out of signatures and cleaning weights, unless the
heading is itself a junk label ("Related posts", "Feedback").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Heading-anchor markup adds a permalink ("¶", "#", a zero-width space)
to the heading text, so "Related posts¶" or "Comments¶" slipped past the
end-anchored junk patterns and their sections were kept. Match the
patterns against the label's words only.
Also treat any label that names reviews as junk, so a product page's
"Ratings and Reviews" section is still removed, and accept an
uppercase one-letter prefix ("H-Social-Tips").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasir-serpapi
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Sites give headings ids made from their own text, so tables of contents can link to them: WordPress uses
h-plus the slug (h-what-is-social-shopping), other sites use the plain slug, sometimes with a counter (travel-related-coverage-2). Documentation generators put the same kind of id on the section a heading opens (Sphinx<section id="cookies">, Django<section id="s-...">).The class/id rules read these ids as the names of sections. "social" in
what-is-social-media-marketing, "sponsor" inh-sponsorship-dealsand "related" intravel-related-coveragematched Boilerplate's noise names, "comment" matched the secondary-content names, and "media" or "widget" gave h1/h2 a negative cleaning weight. Real headings and whole docs sections were dropped. dev/0038 lost every "...Social Media Marketing..." heading, and dev/0662 (Flask docs) lost its "Cookies" section.A survey of main found 77 headings (on 37 pages) that Boilerplate removed only because of an id made from their text: the WCXB reference keeps 67 of them and drops 10.
Change
HeadingAnchors.mark, run once in the Preprocessor, marks an element whose id is made from its heading's text: the heading itself, a wiki headline span inside it, or a section whose first non-empty child is that heading. The comparison ignores case, accents, punctuation and separators, plus a one-letter prefix (h-,s-, also uppercaseH-) and a trailing counter.Dom.signatureandScorer.cleaning_weightleave a marked id out (Dom.name_id). Classes still count. Candidate scoring (Scorer.class_weight) keeps Readability's weight unchanged.NOISE_HEADINGorBOILERPLATE_TEXTmatches ("Related posts", "Share this article", "Comments"), one whose whole text is a furniture or secondary-content name ("Feedback", "Table of contents", "Author bio"), or one that names reviews anywhere ("Customer reviews", "Ratings and Reviews"; reviews are only removed from product pages, where such a section is the reviews widget).<section id="recent-comments-2" class="widget widget_recent_comments"><h2>Recent Comments</h2>) are marked, so only theirwidget_*class names them. WordPress always adds that class, and they are still removed.Results
WCXB, main -> this branch (F1; snippet rates and tokens where they moved):
Regressions
why-posthogis the heading's anchor, and "post" inside "posthog" used to give it a +25 cleaning weight. Without that weight, conditional cleaning removes it. The anchor no longer counts either way.s-be-careful-with-related-name...) is back. The reference leaves out that subsection and "Meta inheritance" but keeps the sections around them.h-glowing-customer-reviews.With #16
git merge-tree --write-tree origin/wcxb-losses HEADreports one conflict, inspec/nokodown/boilerplate_spec.rb. Both PRs add adescribeblock in the same place, after "sections named like furniture". The resolution keeps both blocks: #16's "noise names" first, then "ids made from a heading's text".preprocessor.rb,scorer.rbandnokodown_spec.rbmerge cleanly. The merge'slib/is exactly #16 plus this PR'slib/changes. #16's new rules useDom.signature, so they skip anchors too. All 251 examples pass on the merge.pr16 -> pr16 + this branch (local merge, not pushed):
Scrapinghub 0.9468 -> 0.9468, Mozilla 0.9460 -> 0.9477. The same 36 pages move as on main, with the same single same-level empty heading (dev/0676).
How it was tested
bundle exec rspec: 238 examples, 0 failures. New synthetic specs:heading_anchors_spec.rb: prefixes (also uppercase), counters, wiki ids, accents, apostrophes, a section opener after empty anchor spans, non-anchor ids, junk labels (also "Ratings and Reviews"), and junk labels with a "¶", a "#" before or after, or a zero-width space.lib/, only the 2 "keeps" specs fail (heading_anchors_spec.rbleft out, since the module does not exist there). Run against this PR's first commit (a85f720), the 6 examples that cover the junk-label fix, the uppercase prefix and the reviews section fail.tmp/bench/eval.shagainst main and against pr16. Scrapinghub and Mozilla viabenchmark/quality.rb.🤖 Generated with Claude Code